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DETAILED ACTION 



Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S. C. 1 03(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 



2. Claims 1-21 are rejected under 35 U.S.C. 102(e) as being anticipated over 
Galanes et al. (US PGPub 2003/0200080) in view of Firman (US PGPub 
2002/0010582) and Ativanichayaphong et al. (US PGPub 2004/0236574) 

As to claim 1 , Galanes discloses a method for unifying speech user 
interface and visual prompt on a display (see paragraph [0010]) 
comprising the steps of: receiving grammar specifying a syntax of at least 
one speech command and having semantic information (see paragraph 
[0044]); and processing the grammar to extract the semantic information 
for use with both a graphical user interface and a speech user interface 
(see paragraphs [0004], [0012], and [0044]). 

Galanes doe not specifically disclose receiving speech while 
navigation on the GUI combining the visual and audio, visually presenting 
the speech and building the corresponding commands. Firman teaches 
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the use of a system that unites speech user interface and graphic user 
interface (abstract lines 1-8), receiving a grammar specifying a syntax of a 
command (paragraph [0007] lines 1-6) and processing the grammar to 
extract semantic information (paragraph [0006] lines 1-5). Firman also 
teaches receiving a user input to select at least on component in the 
graphical user interface during a navigation of the graphical user interface 
(paragraph [0004] lines 1-7); audibly receiving the at least one speech 
command associated with the at least one component from the speech 
user interface (paragraph [0004] lines 13-15); and building a 
corresponding speech command for a plurality of speech commands 
associated with a plurality of components (paragraph [0004] lines 15-18). 
Ativanichayaphong teaches a system that unifies visual and audio 
interfaces (abstract lines 1-5) and that allows for the user to enter an audio 
input associated with the object selected for association and visually 
presenting the at least one speech command in a visual prompt of the 
graphical user interface responsive to the step of audibly receiving 
(paragraph [0008] lines 11-15). 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with the receiving of audio input and building commands as taught by 
Firman and the ability to visually display the audio input received from the 
user. Doing so would have allowed for a simple method for the user to 
combine SUI and GUI with out having to use complex programming and 
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confirming the inputs of the commands by visually presenting them would 
have allowed the reduction of errors. 



As to claim 2 and 12, Galanes discloses a step of generating 
graphical user interface elements corresponding to the semantic 
information in the grammar. McNair accomplishes the forming of the GUI 
by the use of audio and visual markup control (see paragraphs [0012] 
lines 1-4; [0049] lines 4-7, and [0051] lines 5-13). 

Galanes does not disclose specifically a single instance of a 
plurality of speech commands. Firman teaches the corresponding speech 
command is a single instance of a plurality of speech commands 
associated with a selection of one or more components of the graphical 
user interface during the navigation of the graphical user interface (a table 
of a plurality of commands are used in order to associate the selected 
command with the selection of a graphical component with navigating 
through the GUI. 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with a single instance of a plurality of speech commands as taught by 
Firma. Doing so would have allowed for organization of how the instances 
where picked and associated with commands that are preprocessed to 
have syntax and meaning. 



V 
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As to claim 3 and 13, Galanes discloses the step of generating 
speech grammar (see paragraph [0051] lines 10-12; [0060] lines 1-3) from 
the grammar (see paragraph [0051] lines 10-12; [0060] lines 18-21) and 
semantic information. The semantic information describes the meaning or 
significance of the information being related between user and multi-modal 
device by way of examples, text, prerecorded audio, audible rendering or 
speech recognition (see paragraph [0060] lines 12-19). 

As to claim 4 and 14, Galanes discloses the method which further 
uses the grammar and the semantic information (like "credit card number") 
to generate visual graphical user interface elements, like labels and 
textboxes, (see paragraph [0061] lines 1-8 and 9-12) required to 
implement a set of commands (like process credit card information) 
contained in the grammar (see paragraph [0061] lines 9 and 10). 

As to claim 5 and 15, Galanes discloses the method wherein the 
step of processing further comprises processing the semantic information 
to generate semantic directives, "meaningful direction", used for parsing 
the grammar between the graphical user interface and the speech user 
interface. Galanes discloses doing this by providing dialog modules (see 
paragraph [0049] lines 4-6) that with the use of markup languages, which 
link graphic and audible controls (see paragraph [0051] lines 1-6), use 
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extensions, tags and other elements that serve as semantic directives to 
unite the GUI and the SUI (see paragraph [0052] lines 1-5). 



As to claim 6 and 21 , Galanes discloses a method for unifying 
speech user interface and graphic user interface commands (see 
paragraph [0010]), comprising the steps of: receiving user entered text via 
a graphical user interface (see paragraph [0061] lines 11-13); processing 
the user entered text via the graphical user interface (see paragraphs 
[0057], [0058]); monitoring the user entered text and adding input context 

to the user entered text (see paragraphs [0057] and [0062]); and updating 

i 

a speech recognizer with the user entered text and semantic information 
of which is done by allowing the user to enter text such as grammar (see 
paragraph [0057]) to be used for audible recognition by the system (see 
paragraph [0062] lines 1-5). 

Galanes doe not specifically disclose receiving speech while 
navigation on the GUI combining the visual and audio, visually presenting 
the speech and building the corresponding commands or a single instance 
of a plurality of speech commands. Firman teaches the use of a system 
that unifes speech user interface and graphic user interface (abstract lines 
1-8), receiving a grammar specifying a syntax of a command (paragraph 
[0007] lines 1-6) and processing the grammar to extract semantic 
information (paragraph [0006] lines 1-5). Firma also teaches monitoring 
the user entered text and adding unput context associated with a selection 
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of at least one component in the graphical user interface to the user 
entered text (paragraph [0005] lines 6-9). Firman also teaches receiving 
a user input to select at least on component in the graphical user interface 
during a navigation of the graphical user interface (paragraph [0004] lines 
1-7); audibly receiving the at least one speech command associated with 
the at least one component from the speech user interface (paragraph 
[0004] lines 13-15); and building a corresponding speech command for a 
plurality of speech commands associated with a plurality of components 
(paragraph [0004] lines 15-18). Ativanichayaphong teaches a system that 
unifies visual and audio interfaces (abstract lines 1-5) and that allows for 
the user to enter an audio input associated with the object selected for 
association and visually presenting the at least one speech command in a 
visual prompt of the graphical user interface responsive to the step of 
audibly receiving (paragraph [0008] lines 11-15). Firman also teaches the 
corresponding speech command is a single instance of a plurality of 
speech commands associated with a selection of one or more 
components of the graphical user interface during the navigation of the 
graphical user interface (a table of a plurality of commands are used in 
order to associate the selected command with the selection of a graphical 
component with navigating through the GUI. 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with the receiving of audio input and building commands and the use of a 



r 
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single instance of a plurality of speech commands as taught by Firman 
and the ability to visually display the audio input received from the user. 
Doing so would have allowed for the for organization of how the instances 
where picked and associated with commands that are preprocessed to 
have syntax and meaning and a simple method for the user to combine 
SUI and GUI with out having to use complex programming and confirming 
the inputs of the commands by visually presenting them would have 
allowed the reduction of errors. 

As to claim 7 and 17, Galanes discloses updating the speech 
recognizer (see paragraph [0045] lines 8-13) by accepting new text 
information and input context (see paragraph [0062] lines 1-5) to augment 
and update a speech grammar and recognition vocabulary of the speech 
recognizer (see paragraph [0045] lines 8-13). 

As to claim 8, Galanes discloses the method wherein the method 
further comprises the step of updating the graphical user interface (see 
paragraph [0058] lines 5 and 6) once the recognizer is updated (see 
paragraph [0057] lines 16-20). 

As to claim 9 and 18, Galanes discloses the method wherein the 
step of updating the graphical user interface (see paragraph [0058] lines 5 
and 6) comprises the steps of updating graphical user interface directives 
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and elements (see paragraph [0052] lines 1-5) to maintain the graphical 
user interface unified with the speech grammar (see paragraph [0051] 
lines 1-6). 

As to claim 10, Galanes discloses the method wherein the method 
further comprises forming a window or multimodal environment (see 
paragraph [0048 lines 13-15) for displaying a speech interface command 
as it is being built using the graphical user interface (see paragraph [0043] 
lines 7-9), in this way the graphic controls can be built with the audible 
control (see paragraph [0060] lines 1-15). 

As to claim 1 1 , Galanes discloses a system having a unified 
speech user interface and visual prompt on a display (see paragraph 
[001 0]), comprising: a display for providing an output for graphic user 
interface elements (see figures 1 and 5); and a processor coupled to the 
display (see paragraph [0026]), the processor programmed to: receive 
grammar specifying a syntax of at least one speech command and having 
semantic information (see paragraph [0044]); and process the grammar to 
extract the semantic information for use with both a graphical user 
interface and a speech user interface (see paragraphs [0004], [0012], and 
[0044]). 

Galanes doe not specifically disclose receiving speech while 
navigation on the GUI combining the visual and audio, visually presenting 
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the speech and building the corresponding commands. Firman teaches 
the use of a system that unites speech user interface and graphic user 
interface (abstract lines 1-8), receiving a grammar specifying a syntax of a 
command (paragraph [0007] lines 1-6) and processing the grammar to 
extract semantic information (paragraph [0006] lines 1-5). Firman also 
teaches receiving a user input to select at least on component in the 
graphical user interface during a navigation of the graphical user interface 
(paragraph [0004] lines 1-7); audibly receiving the at least one speech 
command associated with the at least one component from the speech 
user interface (paragraph [0004] lines 13-15); and building a 
corresponding speech command for a plurality of speech commands 
associated with a plurality of components (paragraph [0004] lines 15-18). 
Ativanichayaphong teaches a system that unifies visual and audio 
interfaces (abstract lines 1 -5) and that allows for the user to enter an audio 
input associated with the object selected for association and visually 
presenting the at least one speech command in a visual prompt of the 
graphical user interface responsive to the step of audibly receiving 
(paragraph [0008] lines 11-15). 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with the receiving of audio input and building commands as taught by 
Firman and the ability to visually display the audio input received from the 
user. Doing so would have allowed for a simple method for the user to 
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combine SUI and GUI with out having to use complex programming and 
confirming the inputs of the commands by visually presenting them would 
have allowed the reduction of errors. 

As to claim 16, Galanes discloses a system having a unified 
speech user interface and graphical user interface (see paragraph [0010]), 
comprising: a display for providing an output for graphic user interface 
elements (see figures 1 and 5); and a processor coupled to the display 
(see paragraph [0026]), the processor programmed to: receive user 
entered text via a graphical user interface (see paragraph [0061] lines 11- 
13); process the user entered text via the graphical user interface (see 
paragraphs [0057], [0058]);monitor the user entered text and adding input 
context to the user entered text (see paragraphs [0057] and [0062]); and 
update a speech recognizer with the user entered text and semantic 
information of which is done by allowing the user to enter text such as 
grammar (see paragraph [0057]) to be used for audible recognition by the 
system (see paragraph [0062] lines 1-5). 

Galanes doe not specifically disclose receiving 
speech while navigation on the GUI combining the visual and audio, 
visually presenting the speech and building the corresponding commands 
or a single instance of a plurality of speech commands. Firman teaches 
the use of a system that unifes speech user interface and graphic user 
interface (abstract lines 1-8), receiving a grammar specifying a syntax of a 
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command (paragraph [0007] lines 1-6) and processing the grammar to 
extract semantic information (paragraph [0006] lines 1-5). Firma also 
teaches monitoring the user entered text and adding unput context 
associated with a selection of at least one component in the graphical user 
interface to the user entered text (paragraph [0005] lines 6-9). Firman 
also teaches receiving a user input to select at least on component in the 
graphical user interface during a navigation of the graphical user interface 
(paragraph [0004] lines 1-7); audibly receiving the at least one speech 
command associated with the at least one component from the speech 
user interface (paragraph [0004] lines 13-15); and building a 
corresponding speech command for a plurality of speech commands 
associated with a plurality of components (paragraph [0004] lines 15-18). 
Ativanichayaphong teaches a system that unifies visual and audio 
interfaces (abstract lines 1-5) and that allows for the user to enter an audio 
input associated with the object selected for association and visually 
presenting the at least one speech command in a visual prompt of the 
graphical user interface responsive to the step of audibly receiving 
(paragraph [0008] lines 11-1 5). Firman also teaches the corresponding 
speech command is a single instance of a plurality of speech commands 
associated with a selection of one or more components of the graphical 
user interface during the navigation of the graphical user interface (a table 
of a plurality of commands are used in order to associate the selected 
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command with the selection of a graphical component with navigating 
through the GUI. 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with the receiving of audio input and building commands and the use of a 
single instance of a plurality of speech commands as taught by Firman 
and the ability to visually display the audio input received from the user. 
Doing so would have allowed for the for organization of how the instances 
where picked and associated with commands that are preprocessed to 
have syntax and meaning and a simple method for the user to combine 
SUI and GUI with out having to use complex programming and confirming 
the inputs of the commands by visually presenting them would have 
allowed the reduction of errors. 

As to claim 1 9, Galanes discloses the system wherein the 
processor is further programmed to form visual prompt on a display 
enabaling controls for speech interface such as speech controls, audible 
prompting, and speech recognition (paragraph [0010] lines 3-5). 

As to claim 20, Galanes discloses a machine-readable storage, 
having stored thereon a computer program having a plurality of code 
sections executable by a machine (see paragraphs [0024]-[0027] and 
[0033]) for causing the machine to perform the steps of: receiving 
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grammar specifying a syntax of at least one speech command and having 
semantic information (see paragraph [0044]); and processing the grammar 
to extract the semantic information for use with both a graphical user 
interface and a speech user interface (see paragraphs [0004], [0012], and 
[0044]). 

Galanes doe not specifically disclose receiving speech while 
navigation on the GUI combining the visual and audio, visually presenting 
the speech and building the corresponding commands or a single instance 
of a plurality of speech commands. Firman teaches the use of a system 
that unifes speech user interface and graphic user interface (abstract lines 
1-8), receiving a grammar specifying syntax of a command (paragraph 
[0007] lines 1-6) and processing the grammar to extract semantic 
information (paragraph [0006] lines 1-5). Firman also teaches receiving a 
user input to select at least on component in the graphical user interface 
during a navigation of the graphical user interface (paragraph [0004] lines 
1-7); audibly receiving the at least one speech command associated with 
the at least one component from the speech user interface (paragraph 
[0004] lines 13-15); and building a corresponding speech command for a 
plurality of speech commands associated with a plurality of components 
(paragraph [0004] lines 15-18). Firman also teaches the corresponding 
speech command is a single instance of a plurality of speech commands 
associated with a selection of one or more components of the graphical 
user interface during the navigation of the graphical user interface (a table 
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of a plurality of commands are used in order to associate the selected 
command with the selection of a graphical component with navigating 
through the GUI. Ativanichayaphong teaches a system that unifies visual 
and audio interfaces (abstract lines 1-5) and that allows for the user to 
enter an audio input associated with the object selected for association 
and visually presenting the at least one speech command in a visual 
prompt of the graphical user interface responsive to the step of audibly 
receiving (paragraph [0008] lines 11-15). 

It would have been obvious to one having ordinary skill in the art at 
the time the invention was made to have modified the method of Galanes 
with the receiving of audio input and building commands and the use of a 
single instance of a plurality of speech commands as taught by Firman 
and the ability to visually display the audio input received from the user. 
Doing so would have allowed for the for organization of how the instances 
where picked and associated with commands that are preprocessed to 
have syntax and meaning and a simple method for the user to combine 
SUI and GUI with out having to use complex programming and confirming 
the inputs of the commands by visually presenting them would have 
allowed the reduction of errors. 
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Conclusion 

A note has been made to notify the appropriate parties that the 
examiner has moved from Art Unit 2609 to 2626. 

Any inquiry concerning this communication should be directed to 
Josiah Hernandez whose telephone number is 571-270-1646. The 
examiner can normally be reached from 7:30 pm to 5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, 
the examiner's supervisor, David Hudspeth can be reached on (571) 272- 
7843. The fax phone number for the organization where this application 
or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained 
from the Patent Application Information Retrieval (PAIR) system. Status 
information for published applications may be obtained from either Private 
PAIR or Public PAIR. Status information for unpublished applications is 
available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on 
access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). 
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